Individuals interact with computers through a user interface. The user interface enables a user to provide input to and receive output from the computer. The output provided can take on many forms and often includes presenting a variety of user-interface elements, sometimes referred to as “controls.” Exemplary user-interface elements include toolbars, windows, buttons, scrollbars, icons, selectable options, and the like. Virtually anything that can be clicked on or given the focus falls within the scope of “element” as used herein. Information related to user-interface elements is often requested by assistive-technology products so that the products can enhance a user's computing experience.
Assistive-technology products are specially designed computer programs designed to accommodate an individual's disability or disabilities. These products are developed to work with a computer's operating system and other software. Some people with disabilities desire assistive-technology products to use computers more effectively.
Individuals with visual or hearing impairments may desire accessibility features that can enhance a user interface. For example, individuals with hearing impairments may use voice-recognition products that are adapted to convert speech to sign language. Screen-review utilities make on-screen information available as synthesized speech and pairs the speech with visual representations of words in a format that assists persons with language impairments. For example, words can be highlighted as electronically read. Screen-review utilities convert text that appears on screen into a computer voice.
To provide supportive features to persons that desire to use them, assistive-technology applications do not have access to the same code that native applications are able to use. This is because an assistive-technology application works on behalf of a user; instead of the user working directly with the user interface—as is the case in native applications. For instance, if a word-processing application wishes to display text to user, it can easily do so because the word-processing application knows what program modules to call to display the text as desired. But a screen reader—an application that finds text and audibly recites the text to a user—is unaware of much of a target application's programmatic code. The screen reader must independently gather the data needed to identify text, receive it, and translate it into audio.
Assistive-technology applications work under a variety of constraints. To further illustrate a portion of the constraints that assistive-technology applications are subject to, consider, for example, an application that needs to display the contents of a listbox. This would be an easy task for a native application because it would know where the relevant list-box values are stored and simply retrieve them for display. But an assistive-technology application does not know where the values are stored. It must seek the values itself and be provided with the necessary information to display the values. Thus, assistive-technology applications must function with limited knowledge of an application's user interface.
The difficulties associated with an assistive-technology application performing certain functions on all types of user-interface elements is somewhat akin to the difficulties that would be faced by a person asked to be able to program any type of VCR clock simply by providing access to the VCR clock. Unlike the VCR owner who is familiar with his VCR's clock and has the VCR manual, the fictitious person here has no foreknowledge of what type of VCR he may come across, what type of actions are necessary to program the clock, whether it will be a brand ever seen before, or the means of accessing its settings—which may be different from every other VCR previously encountered. Moreover, expecting the person to know about every type of VCR is an unrealistic proposition. As applicable to the relevant art, it is an unrealistic proposition to expect every client application to know about every type of listbox that it might encounter. Programming such a client application would be an expensive and resource intensive process.
One way a user interface may provide this information is by using logical hierarchal structures. A significant problem in the art, however, is that logical hierarchal structures provided by a user interface often do not have the requisite level of granularity needed by an assistive-technology application. Without the benefit of an adequate description of a UI or knowing the contents of certain data elements (such as listboxes, combo boxes, and many others), assistive-technology applications must request this information from the user interface to be able to manipulate or otherwise make use of the data.
Although assistive-technology applications can provide various user-interface customizations if they can receive accurate data regarding the user-interface elements, providing accurate information regarding user-interface elements has proven difficult. This difficulty stems from the fact that no single entity knows all the relevant information about any particular piece of a user interface. For example, although a list-box component may itself know the individual list-box items contained within it, only the name of the listbox may be known by its parent dialog window. Although a user interface or portion of a user interface may be depicted as a hierarchal structure such as a tree, a single tree may only provide limited information, which can prevent an assistive-technology application from functioning properly.
A shortcoming exists in the current state of the art whereby providing information described by two or more logical trees is either impossible or inordinately difficult. For example, a user interface may have three windows with a button and a listbox in one of the three windows. Information about the user interface may be contained in a first tree while information about the contents of the listbox may be described by a second tree. In such a situation, an assistive-technology application that requires knowledge of both trees must try to derive this information itself, which is difficult. There is no efficient way to represent the two or more trees to the client application as a single hierarchal representation. Accordingly, there is a need for a method and system for providing accurate, comprehensive hierarchal-structure information about user-interface components described in two or more logical trees to a requesting application. Moreover, there is a need to provide the information in a format that is easy for the requesting application to process.